Block use of internal and external snapshots on KVM #11039

JoaoJandre · 2025-06-16T18:37:01Z

Description

On KVM, there are two types of snapshots: internal and external. Most snapshot/backup solutions use external snapshots on ACS; save for disk-and-memory VM snapshots, which use internal snapshots (this is a limitation with KVM, as far as I know).

However, since internal snapshots are stored inside the VM's volume (hence the name), if an internal snapshot is taken after an external snapshot and the external snapshot is restored, the internal snapshot is lost.

Thus, this PR blocks the use of disk-and-memory VM snapshots alongside volume snapshots, NAS backups, and disk-only VM snapshots (at least the ones created using the default volume snapshot implementation).

I encourage maintainers of 3rd party storage providers to test if their implementation is compatible with disk-and-memory VM snapshots, if it is not it their simultaneous usage should be blocked.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

I created a VM and created a few disk-and-memory VM snapshots on it; then I tried to create NAS backups, volume snapshots and disk-only VM snapshots, all of them gave me an error, which is expected.

I validated that the opposite was also true for the aforementioned cases, e.g., create volume snapshot and try to create disk-and-memory VM snapshot.

I also validated that it was possible to create multiple NAS backups, disk-only VM snapshots and volume snapshots with no issues.

JoaoJandre · 2025-06-16T18:39:18Z

@slavkap @rp- I think it would be interesting to validate if the implementations done for Storpool and Linstor are compatible with disk-and-memory VM snapshots.

codecov · 2025-06-16T18:44:53Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 3.63%. Comparing base (574ed78) to head (8938966).
⚠️ Report is 6 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (574ed78) and HEAD (8938966). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (574ed78) HEAD (8938966)

unittests 1 0

Additional details and impacted files

@@              Coverage Diff              @@
##               main   #11039       +/-   ##
=============================================
- Coverage     17.36%    3.63%   -13.73%     
=============================================
  Files          5888      441     -5447     
  Lines        525737    37019   -488718     
  Branches      64164     6785    -57379     
=============================================
- Hits          91274     1345    -89929     
+ Misses       424163    35513   -388650     
+ Partials      10300      161    -10139

Flag	Coverage Δ
uitests	`3.63% <ø> (-0.01%)`	⬇️
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

JoaoJandre · 2025-06-16T18:48:48Z

@blueorangutan package

blueorangutan · 2025-06-16T18:50:04Z

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-16T19:10:16Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13798

blueorangutan · 2025-06-17T07:45:56Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13809

DaanHoogland · 2025-06-17T08:43:04Z

@JoaoJandre :

08:35:48 [ERROR] /jenkins/workspace/acs-centos8-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.2.0-SNAPSHOT/engine/storage/snapshot/src/test/java/org/apache/cloudstack/storage/vmsnapshot/VMSnapshotStrategyKVMTest.java:32:8: Unused import - org.apache.cloudstack.backup.dao.BackupDao. [UnusedImports]

JoaoJandre · 2025-06-17T16:38:53Z

@blueorangutan package

blueorangutan · 2025-06-17T16:40:04Z

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-17T17:02:31Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13818

DaanHoogland · 2025-06-18T07:04:45Z

@blueorangutan package

blueorangutan · 2025-06-18T07:08:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-18T08:16:00Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13826

rp- · 2025-07-03T08:09:10Z

Linstor does currently not support memory snapshots (we check and throw an error if selected).
So I guess we are currently not affected by any of this?

...apshot/src/main/java/org/apache/cloudstack/storage/vmsnapshot/DefaultVMSnapshotStrategy.java

slavkap

code LGTM
I haven't tested it with NFS, but the StorPool smoke tests are executed successfully

DaanHoogland · 2025-07-08T15:00:32Z

@blueorangutan test

blueorangutan · 2025-07-08T15:02:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-07-09T06:43:28Z

[SF] Trillian test result (tid-13723)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53428 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11039-t13723-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

weizhouapache · 2025-08-28T13:23:15Z

@JoaoJandre
this seems to be included in PR #10632
do you still want it in 4.20.2 ?

JoaoJandre · 2025-08-28T13:31:46Z

@JoaoJandre this seems to be included in PR #10632 do you still want it in 4.20.2 ?

@weizhouapache PR #10632 blocks the usage of the feature introduced in #10632 and other incompatible features. This PR purposefully ignores #10632 and adds restrictions to avoid other interactions between internal and external snapshots; such as volume snapshot and disk-and-memory VM snapshot.

They are complementary. When merging this PR forward, care should be taken so that the validations of both PRs do not erase one another (I can make the merge forward if needed).

weizhouapache · 2025-08-28T13:36:51Z

@JoaoJandre this seems to be included in PR #10632 do you still want it in 4.20.2 ?

@weizhouapache PR #10632 blocks the usage of the feature introduced in #10632 and other incompatible features. This PR purposefully ignores #10632 and adds restrictions to avoid other interactions between internal and external snapshots; such as volume snapshot and disk-and-memory VM snapshot.

They are complementary. When merging this PR forward, care should be taken so that the validations of both PRs do not erase one another (I can make the merge forward if needed).

ok @JoaoJandre
I think the best option might be re-target this PR to 4.22 which includes #10632 , to avoid re-work.

DaanHoogland · 2025-08-29T07:36:17Z

aren't we talking 4.20.2 , @weizhouapache ?

weizhouapache · 2025-08-29T07:44:25Z

aren't we talking 4.20.2 , @weizhouapache ?

sorry, I meant 4.22, not 4.21

if we merge into 4.20.2, the merge forward to 4.22 will be a trouble , as @JoaoJandre mentioned
unless we ignore this PR in merge forward, and @JoaoJandre create another PR against 4.22 (needs re-review and re-testing)

JoaoJandre · 2025-09-01T18:07:47Z

@DaanHoogland @weizhouapache I rebased the changes so now I'm targeting main.

...apshot/src/main/java/org/apache/cloudstack/storage/vmsnapshot/DefaultVMSnapshotStrategy.java

weizhouapache · 2025-09-01T18:21:40Z

@JoaoJandre
thanks for the update, overall LGTM.
left a small comment

weizhouapache

code lgtm

not tested

boring-cyborg bot added component:backup component:storage labels Jun 16, 2025

JoaoJandre added the Severity:Major label Jun 16, 2025

JoaoJandre added this to the 4.20.2 milestone Jun 16, 2025

DaanHoogland requested review from rp-, slavkap and harikrishna-patnala June 17, 2025 06:17

slavkap reviewed Jul 8, 2025

View reviewed changes

...apshot/src/main/java/org/apache/cloudstack/storage/vmsnapshot/DefaultVMSnapshotStrategy.java Outdated Show resolved Hide resolved

slavkap reviewed Jul 8, 2025

View reviewed changes

...apshot/src/main/java/org/apache/cloudstack/storage/vmsnapshot/DefaultVMSnapshotStrategy.java Outdated Show resolved Hide resolved

slavkap approved these changes Jul 8, 2025

View reviewed changes

Avoid data loss

f54062d

JoaoJandre force-pushed the avoid-data-loss branch from 0269142 to f54062d Compare September 1, 2025 18:04

boring-cyborg bot added component:agent component:api component:build component:integration labels Sep 1, 2025

JoaoJandre changed the base branch from 4.20 to main September 1, 2025 18:05

weizhouapache reviewed Sep 1, 2025

View reviewed changes

...apshot/src/main/java/org/apache/cloudstack/storage/vmsnapshot/DefaultVMSnapshotStrategy.java Outdated Show resolved Hide resolved

address review

8938966

weizhouapache approved these changes Sep 3, 2025

View reviewed changes

weizhouapache modified the milestones: 4.20.2, 4.22.0 Sep 3, 2025

Block use of internal and external snapshots on KVM #11039

Are you sure you want to change the base?

Block use of internal and external snapshots on KVM #11039

Uh oh!

Conversation

JoaoJandre commented Jun 16, 2025

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

JoaoJandre commented Jun 16, 2025

Uh oh!

codecov bot commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JoaoJandre commented Jun 16, 2025

Uh oh!

blueorangutan commented Jun 16, 2025

Uh oh!

blueorangutan commented Jun 16, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

DaanHoogland commented Jun 17, 2025

Uh oh!

JoaoJandre commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

DaanHoogland commented Jun 18, 2025

Uh oh!

blueorangutan commented Jun 18, 2025

Uh oh!

blueorangutan commented Jun 18, 2025

Uh oh!

rp- commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

slavkap left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Jul 8, 2025

Uh oh!

blueorangutan commented Jul 8, 2025

Uh oh!

blueorangutan commented Jul 9, 2025

Uh oh!

weizhouapache commented Aug 28, 2025

Uh oh!

JoaoJandre commented Aug 28, 2025

Uh oh!

weizhouapache commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DaanHoogland commented Aug 29, 2025

Uh oh!

weizhouapache commented Aug 29, 2025

Uh oh!

JoaoJandre commented Sep 1, 2025

Uh oh!

Uh oh!

weizhouapache commented Sep 1, 2025

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Jun 16, 2025 •

edited

Loading

weizhouapache commented Aug 28, 2025 •

edited

Loading